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ARt  Research  Reports  and  Technical  Papers  are  intended  for  sponsors  of 
R&O  tasks  and  other  research  and  military  agencies.  Any  findings  ready  for 
implementation  at  the  time  of  publication  are  presented  •»*  tr.c  latter  part  of 
the  Brief.  Upon  completion  of  a  major  phase  of  the  task,  formal  recommen¬ 
dations  for  official  action  normally  are  conveyed  to  appropriate  military 
agencies  by  briefing  or  Disposition  Form. 


FOREWORD 


The  Leadership  Performance  Technical  Area  ‘^as  among  its  objectives  the  identification  of 
personal  characteristics  of  performance  and  their  potential  for  use  in  the  Officer  Career 
Management  System  and  th-.  points  of  application  of  these  measures  in  improving  current  pro¬ 
cedures  used  for  military  school  selection,  promotion  nomination,  and  duty  assignment.  As  one 
aspect  of  ihe  evaluation  of  these  personal  characteristics,  research  is  now  underway  'or  the  experi¬ 
mental  introduction  of  peer  ratings  in  Officer  Basic  Courses  and  is  projected  for  Officer  Advanced 
Courses.  Peer  {or  associate)  evaluations  have  in  the  past  been  found  to  be  valid  predictors  of 
future  performance  (potential)  in  a  number  of  military  situations  but  must  be  investigated  in  the 
new  setting  in  which  they  are  now  being  applied. 

The  entire  task  is  responsive  to  special  requirements  of  the  Deputy  Chief  of  Staff  for 
Personnel  and  the  Military  Personnel  Center,  as  well  as  to  the  objectives  of  ROTE  Project 
2Q16310A755,  Manpower  System  Development. 

The  present  publication  examines  the  effects  of  evaluation  procedures  on  psychometric 
properties,  reliability,  and  concurrent  validity  of  associate  evaluations. 
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BRIEF 


REQUIREMENT- 

To  determine  if  a  nomination  procedure  of  associate  evaluations  can  be  substituted  for  a  rating 
procedure. 


PROCEDURE: 

Data  were  collected  on  125  Army  officers  attending  Branch  Basic  School.  Three  different 
scoring  procedures  w.-re  used,  representing  a  rating  procedure  and  two  nomination  procedures. 
Estimations  of  reliabilities  were  compared  across  procedures,  and  correlations  (indices  of 
relationships)  were  compared  with  a  degree  of-acquaintanceship  scare,  a  Leadership  Battery,  and 
school  grades. 


FINDINGS: 

The  reliabilities  of  ali  procedures  were  very  similar,  with  some  indications  that  the  use  of  too 
many  individuals  in  a  nominations  technique  might  lower  reliability.  With  the  exception  of  the 
acquaintanceship  scores,  there  were  no  differences  between  techniques  in  the  correlation  with 
other  scores.  The  nomination  technique  with  fewer  individuals  nominated  had  a  significant^  lower 
relationship  with  acquaintanceship.  A  nomination  procedure  is  found  most  readily  usable. 


UTILIZATION  OF  FINDINGS: 

The  present  analysis  is  the  first  step  in  the  experimental  introduction  of  associate  ratings  into 
Army  Schools.  The  use  of  a  nomination  technique  saves  rater  time  and  effort,  is  administratively 
simple  and  increases  acceptance  ot  associate  evaluations.  Future  research  will  focus  upon  the  issues 
of  reliability  across  schools,  acceptability,  feasibility,  and  validity  cf  associate  evaluations. 
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ASSOCIATE  EVALUATIONS:  NOMINATIONS  VS.  RATINGS 


THE  PROBLEM 

The  U.S.  Army  has  a  long  history  of  using  associate  evaluations-- 
pc jr  ratings--as  selection  and  evaluation  devices  in  training  situations. 
The  best  known  and  most  comprehensively  researched  Array  program  is  the 
"Aptitude  for  Service  Ratings"  at  the  U.  S.  Military  Academy.1  2 
Associate  evaluations  have  also  been  investigated  for  use  in  personnel 
selection.  Peer  ratings  in  industry  have  been  found  capable  of  predic¬ 
ting  future  performance  as  well  as  accurately  reflecting  current  perfor¬ 
mance.3  However,  the  methods  used  in  obtaining  the  data  have  varied 
considerably.  Two  methods  are  used  most  frequently:  the  rating  procedure, 
where  each  member  of  the  rating  group  assigns  every  other  member  a  score 
from  an  evaluation  scale,  and  the  nomination  procedure,  where  each 
member  of  the  rating  group  selects  a  given  number  of  top  and  bottom 
individuals  in  terms  of  value  from  the  total  group.  While  both  techniques 
have  produced  similar  results,  they  have  been  used  primarily  in  different 
studies,  so  that  direct  comparisons  are  limited.  However.  Suci4  found 
that  several  procedures  produced  results  of  about  the  same  reliability 
and  recommended  the  use  of  nominations  because  they  were  easier  to 
prepare,  administer,  and  score  and  less  frustrating  to  the  rater. 

Hammer6  also  found  that  rankings  and  nominations  of  the  same  individuals 
produced  similar  evaluations  and  recommended  the  use  of  nominations. 
However,  neither  studied  the  differences  between  the  two  techniques  in 
their  relationship  with  other  measures. 


1Haggerty,  H.F.  Status  report  on  research  for  the  U.S.  Military  Academy. 
ARI  Technical  Research  Report  1155-  (DDC  4J2  090)  October  1963. 

2 

Tobin,  D.J.,  and  Marcum.  R.H.  Leadership  evaluation.  Research  Report, 
Office  of  Military  Psychology  and  Leadership,  U.S.  Army  Military  Academy. 
West  Point,  N.Y.,  I967. 

3 

Nadal,  Ramon  A.  A  review  of  peer  rating  studies.  Research  Report  No. 
68-8,  Office  of  Military  Psychology  and  Leadership,  U.  S.  Amy 
Military  Academy.  West  Point,  N.Y.  ,  I968. 

♦Suci,  G.J. .  Vailance,  T.R.  and  Glickman,  A.S.  An  analysis  of  peer 
ratings:  I.  The  assessment  of  reliability  of  several  question  forms 
and  techniques  used  at  the  Naval  Officer  Candidate  School.  Bureau  of 
Naval  Personnel  Technical  Bulletin  54-9.  Newport,  R.I.,  1954. 

faHammer,  C.H,  A  simplified  technique  for  evaluating  basic  trainees  on 
leadership  potential.  ARI  Research  Memorandum  63-10.  1963» 

-/* 


OBJECTIVES 


The  overall  objectives  of  thi3  study  were  to  Investigate  the  reli¬ 
abilities  of  three  types  of  associate  evaluations  and  further  to  compare 
each  evaluation's  relationships  with  other  measures  of  leadership  and 
school  performance.  The  specific  objectives  of  the  research  were  to 
compare  three  types  of  associate  evaluations  scoring--one  rating  proce¬ 
dure  and  two  nomination  procedures  —  in  terms  of  1)  reliability,  2) 
interrelationship  of  associate  evaluation  techniques,  3)  relationship 
with  other  leadership  measures  (i.e.,  th<  Officer  Evaluation  Battery), 
and  4)  relationship  with  concurrent  performance  measures  (i.e.,  school 
grades).  Points  J)  &  4)  were  included  to  expand  knowledge  beyond 
results  of  previous  studies. 


METHOD 


Sample  Population 

All  officers  attending  a  nine  week  training  course  (N  =  125)  were 
used  for  the  study.  Almost  all  were  2nd  Lieutenants  on  active  duty  only 
for  the  training  period.  While  some  individuals  were  appointed  on  a 
rotating  tasis  to  student  command  positiors,  these  positions  were  pri¬ 
marily  nominal  in  nature.  The  officers  attended  classes  approximately 
8  hours  a  day  five  days  a  week  and  went  "home"  in  the  evening.  The 
officers  were  split  into  4  platoons  the  platoon  being  the  evaluation 
group  within  which  leadership  choices  were  made.  Once  an  associate 
evaluation  score  was  produced  within  a  rating  group  then  all  individuals 
from  all  platoons  were  combined  into  one  group  for  analyses  reported  in 
the  results  section. 


Variables 

Four  distinct  sets  of  date  were  collected.  They  were: 

Associate  evaluations.  Each  officer  was  forced  to  rate  individuals 
along  a  7-point  scale  with  "equal"  numbers  in  each  category  (rating 
scale).  Each  officer  was  then  instructed  to  select  the  one  officer  in 
his  platoon  who  had  the  highest  leadership  potential.  Next  he  was 
instructed  to  select  the  officer  who  had  the  lowest  leadership  potential, 
continuing  until  1/7  of  the  group  was  in  the  high  and  1/7  in  the  low 
categories  (nomination  score  1).  He  then  continued  with  the  next  highest 
and  lowest  1/7  (top  and  bottom  two  categories,  nomination  score  2)  and 
again  the  highest  and  lowest  1/7;  the  remaining  I/7  was  placed  in  a 
middle  category  and  included  the  individuals  he  did  not  know. 

Experimental  diagnostic  leadership  measures.  The  Officer  Evaluation 
Battery  (OEB)  (PT  4934  and  FT  4955)  was  administered  to  all  officers  at 
the  start  of  training.  The  OEB  yields  seven  scale  scores: 
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Combat  Leadership,  Technical-Managerial  Leadership,  and  Career  Potential 
with  a  cognitive  (or  knowledge  factor)  and  a  non-cognitive  (or  attitu- 
dinal  factor)  for  each,  plus  Career  Intent.  (See  the  "Manual  for  Inter¬ 
preting  the  Officer  Evaluation  Battery"  6  for  further  explanation  of  the 
scales  and  development  of  the  test.) 

Training  grades.  A  variety  of  evaluative  techniques  were  used  by 
the  school  to  measure  performance  during  training.  Table  1  lists  the 
various  evaluations  used. 

Acquaintanceship  ratings.  Each  officer  was  instructed  to  rate  the 
degree  to  which  he  was  acquainted  with  each  member  of  his  group.  Ratings 
were  done  on  a  five-point  scale  (1  =  DO  NOT  KNOW;  2  =  MET  ONCE  OR  TWICE; 

3  =  LIMITED  CONTACT  IN  CLASSES;  4  =  EXTENSIVE  CONTACT  IN  CLASSES;  and 
5  =  CLOSE  PERSONAL  RELATIONSHIP). 

Analysis 

Peer  evaluations  were  scored  three  different  ways. 


1)  R  =  z"  r 

n 

2)  N1  =  s!J  ^ 

n 

3)  Ng  -  tTg 

n 

wh:*re: 

R  =  associate  rating  score 
r  =  scale  score  (1-7)  received  by  a  person 
n  =  number  of  persons  giving  an  evaluation 
=  nomination  score  1 

rT^  *  scale  score  transformed  as  follows: 

1  =  1;  2,  3,  4,  5,  or  6  =2;  and  7=3 
N^  =  nomination  score  2 
^2  =  scale  score  transformed  as  follows: 

1  or  2  =  1;  3,  4,  or  5  =  2;  and  6  or  7=3 


“U.S.  Army  Research  Institute  tor  the  Behavioral  and  Social  Sciences. 
Manual  for  interpreting  the  Officer  Evaluation  Battery.  Arlington.  Va: 
Army  Research  Institute, 
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Table  1 


MEANS  AND  STANDARD  DEVI  AT  It 'NS  OF  VARIABLES 
(N  =  125) 


Variable 

X 

Standard  Deviation 

Evaluations 

Retting  (R)  4th  week 

5.9.’ 

1.18 

Rating  (R)  8th  week 

5-96 

1.16 

Nomina t  .on  (N, )  4th  week 

(top  t;  bottom  categories) 

2.00 

.30 

Nomination  (N., )  8th  veek 

2.00 

.20 

Nomination  (N_)  4th  w.ek 

(top  &  bottom  two  categories) 

2.12 

.56 

Nomination  (N^)  6th  wer.k 

2.11 

.36 

4ch  week  acquaintanceship  r-iting 

3*37 

•36 

8th  week  acquaintanceship  rating 

'*5 

•34 

OEB 

Combat  Leadership  (Cognitive) 

107.62 

20.07 

Managerial-Tech  L*ad.  (Cognitive) 

107.27 

21.80 

Career  Potential  (Cognitive) 

109.44 

20.73 

Combat  Leadership  (Non-eognitive) 

109.05 

Managerial-Tech  Lead,  (Non-cognitive ) 

101.47 

in 

Career  Potential  (Non-cognitive) 

103-93 

16.20 

Career  Intent  (Non-cognitive) 

93.39 

17.66 

School  Grades 

Maintenance  Management 

84.12 

8.08 

Combat  Engineer  Practical  (Lead.) 

85.05 

8.24 

Leadership  Exam 

77.45 

10.13 

Night  Land  Navigation 

93.60 

15.64 

Rjysical  Fitness 

79-68 

5.2” 

Leadership,  Staff,  Intelligence,  etc. 

89.50 

7.^3 

Combat  Operation 

80.99 

,.64 

Engineer  Reconnaissance 

85.86 

7.89 

Combat  Engineer  Practical  (Tech) 

80.48 

6.50 

Orienteering 

89.12 

7.52 

Fixed  Bridges  and  Construction 

80.04 

12.64 

Heavy  Construction 

79.83 

15.83 

Acquaintanceship  ratings  were  converted  to  scores  for  an  indivual  by 
computing  his  mean  ratings.  His  score  then  became  the  degree  to  which 
he  was  known  by  the  group  as  a  whole. 

Reliabilities  of  associate  evaluations  were  estimated  by  using  the 
split-half  (group)  technique,7  where  random  halves  (raters)  of  the  rating 
group  were  used  to  produce  two  separate  scores  for  each  individual. 

These  two  scores  from  each  half  were  correlated  with  each  other  over  all 
rating  groups.  The  correlation  was  then  corrected  by  use  of  the  Spearman- 
Brown  prophecy  formula.  The  same  split  of  the  rating  group  was  used  for 
the  split -half  estimate  for  all  three  associate  evaluation  techniques. 

Product  moment  correlations  were  computed  between  all  associate 
evaluation  techniques.  Zero  order  correlations  were  computed  between 
each  associate  evaluation  technique  and  the  remaining  variables. 
Hotelling's  t-tests8  for  differences  between  pair-wise  correlation 
coefficients  were  performed. 


RESULTS 

Table  1  lists  all  variables  with  their  means  and  standard -deviations. 
Table  2  presents  the  split-half  reliabilities  and  intercorrelations  for 
all  associate  evaluation  techniques  and  for  4th  and  8th  weeks.  The 
split-half  reliabilities  were  quite  similar  for  Ratings  and  nomina¬ 
tions  (top/bottom  categories),  but  Ng  Nominations  (top/bottom  two 
categories)  were  lower.  .85.  The  test-retest  reliabilities  were  very 
high  (.90's)  for  all  methods.  Finally,  there  was  a  high  degree  of 
relationship  between  all  techniques  for  one  session  with  slightly 
smaller  values  across  sessions. 

The  relationships  of  associate  evaluations  with  the  Officer  Evalua¬ 
tion  Battery  and  school  grades  are  shown  in  Table  3.  One  hundred  and 
twenty-six  pair-wise  Hotelling  t-tests  were  computed  for  differences 
between  correlation  coefficients  for  each  associate  evaluation  (R  N- 
and  N_  for  the  fourth  and  the  eighth  week);  i.e.,  six  t-tests  (R  vs  N 
R  vs  Ng,  and  vs  Ng  for  the  fourth  and  eighth  weeks)  were  performed^*’ 
for  each  of  the  21  variables.  Three  significant  differences  (p  <  .05) 
were  found  (t  *  2.35  for  fourth  week  R  vs  N?,  Managerial-Technical 
Leadership,  cognitive;  t  =  2.29  for  eighth  week  R  vs  N„  Managerial-Tech¬ 
nical  Leadership,  mra-cognitive;  and  t  *  2.67  for  fourth  week  R  vs  N. 
Fixed  Bridges  and  Construction  exam).  It  was  recognized  that 


G-rdon,  L.V.  Estimating  the  reliability  of  peer  ratings.  Educational 
and  Psychological  Measurement.  I969,  29,  305-313.  . . 

8 Guilford ,  J.P.  Fundamental  statistics  in  Psychology  and  Education 
(4th  ed.)  New  York: McGraw-Hill,  I956.  . 
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CORRELATION  OF  ASSOCIATE  EVALUATIONS 
WITH  OEB  AND  SCHOOL  GRADES 

(N-125) 


Heavy  Construction 


the  three  tests  performed  within  a  ueei  p  u  „ 

2hS  3  5.” 

££.*,2:  r.uc”^rlr»«gn“i“,;t  " 

the  results.  ‘  Pr°duCe  an  ide"tifiable  distortion  in 

Table  4^ f ^No^ignif icant^iff6^  frOW  ac<JuaintanceshiP,  as  is  shown  in 

N2  Nominat  ions ^but  ^'ominat ions^aH6^  ^  betW~"  *•«*»«■  and 

with  acquaintanceship^han  either  Rating  8lgnJfi*;antly  lower  relationship 

4th  week:  t  «  3.64  V  <  S vf  t  **0?  *2  Noai^tio08  (R  vs  Nlt 

*  »  »J.  8th  „.e\: V-  ls?’J f«rek:„  *  ;  3-05,  P  <  *.05; 

t  »  5.16,  p  <  tQs)  '  *  "  ’  3nd  N1  vs  N 2 ,  8th  week: 


Table  4 


CORRELATION  OF  ASSOCIATE  EVALUATIONS  WITH  ACQUAINTANCESHIP 


Acqua intanceshi p 

Evaluation 

Fourth  week 

Eighth  week 

Rating  4th  week 

•  G4 

.51 

Rating  8th  week 

.65 

-50 

Nomination  (N^)  4th  week 

.54 

•  33 

Nomination  (Nx)  8th  week 

*  ✓  1 

•39 

Nomination  (Ng)  4th  week 

.65 

•55 

Nomination  (Ng)  8th  week 

.€1 

•53 

wwnvkVWIVHO 


The  three  methods  of  scoring  the  associate  evaluations  yielded 

;^T  %  eLCl81°f  feliabllity  which  ««e  high  enough  to  justify  their 
use  for  individual  selection  purposes.  There  was  some  indication  that 

fNC  ZilV!tg!  numbers  of  individuals  in  high  and  low  categories 
(N?  Nomination  technique)  might  yield  slightly  lower  split-half 
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reliabilities.  Tnis  would  seem  to  reflect  the  difficulty  of  tasking 
reliable  discrimination  for  the  category  6  and  2  individuals  plus  the 
dilution  of  this  information  into  category  ?  atid  1  individuals.  Further¬ 
more,  the  correlations  of  each  technique  with  other  leadership  measures 
(OEB  and  school  grades)  and  technical  training  grades  indicated  that  the 
associate  evaluation  techniques  were  all  measuring  the  same  things. 

This  was  further  substantiated  by  the  high  degree  of  interrelationship 
between  techniques.  The  only  difference  found  was  the  lover  degree  of 
relationship  between  Ni  Nominations  and  acquaintanceship,  scores.  The 
implication  is  that  the  less  extreme  scores  (middle  categories)  are 
determined  more  by  the  degree  to  which  s  person  is  known  by  individuals 
in  the  group.  This  did  not,  however,  affect  the  relationship  with  other 
measures.  This  finding  of  a  lack  of  relationship  between  acquaintance¬ 
ship  and  performance  has  been  consistent3. 

On  the  basis  of  these  findings  it  would  seem  that  nominations,  using 
a  relatively  small  number  of  individuals,  can  be  substituted  for  full 
rating  without  any  loss  in  reliability  or  degree  of  'elationship  with 
concurrent  performance  measures.  A  r^tential  benefit  is  a  decreased 
reliance  on  acquaintanceship  (frie^dship/popularity)  for  the  more 
difficult  middle  category  evaluations. 

An  assumption  made,  but  yet  unproven,  is  that  modifying  the  instruc¬ 
tions  to  the  raters  to  reflect  only  a  nomination  technique  will  not 
change  their  behavior,  i.e. ,  the  individual",  selected.  Research  is  now 
underway  to  investigate  the  results  of  using  a  nomination  technique 
very  similar  to  the  one  administered  here  but  with  instructions  for 
choosing  individuals  for  only  the  top  and  bottom  categories. 

If  the  additional  benefits  of  decreased  rater  resistance  so  making 
nominations  and  the  greater  ease  with  which  evaluations  can  be  adminis¬ 
tratively  handled  and  scored  are  added  to  the  above  research  findings,  the 
nominations  (N^)  technique  is  the  clear  choice  for  operational  use.  Two 
cautions  should  be  added  to  this  generalization.  First,  the  effect  of 
group  size  was  not  investigated  and  there  are  some  reasons  to  suspect 
that  the  findings  might  not  hold  for  smaller  evaluation  groups.  Second, 
the  use  of  associate  evaluations  as  measures  of  long-term  performance 
was  not  studied  and  the  possibility  exists  that  the  evaluation  techniques 
could  yield  different  results  for  these  measures.  These  two  potential 
problem  areas  are  now  under  investigation. 


Hollander,  E.P.  and  Webb,  W.B.  Leadership,  followship,  and  friendship, 
an  analysis  of  peer  nominations.  Journal  of  Abnormal  and  Social- 
Psychology  .  I955,  20,  163-16?. 
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